00100	.turn on "|{}"
00200	.macro figure(n)⊂
00300	.skip once CENTER
00400	Figure  n  about here
00500	.skip ⊃
00600	
00700	.skip 2 once flush left select 1
00800	%3 1. INTRODUCTION.%*
00900	.indent 5
01000	
01100	Recent years have seen a greatly increased level of conversation between
01200	students of psychology, and of artificial intelligence(AI). This increase
01300	seems to stem, on the one hand, from AI's partial acquiescence to the notion
01400	that psychological evidence ,
01500	particularly as couched in the models due to Neisser(Neisser,1967),
01600	Norman(Norman,1970) and
01700	others,
01800	may have a place in its own thinking about cognitive processes,
01900	and on the other, from a spreading appreciation on the
02000	part of psychologists of the potential virtues of the computational metaphor.
02100	This latter movement has been  particularly evident in the literature on
02200	memory structures(e.g.Anderson & Bower,1973), but seems as yet not to have had
02300	 significant impact
02400	upon studies of the perceptual process. That the study of the mechanisms of
02500	perception of real-world scenes (which we sharply distinguish from the 
02600	reading process) is not widespread in experimental psychology at the present
02700	moment seems partly to be a matter of fashion and, more importantly, to be due to
02800	the apparent unavailability of powerful information-processing concepts on a 
02900	par with those concerning the organization of memory.
03000	
03100	In this essay, we wish to suggest that recent advances in the construction of
03200	artificial vision systems provide the beginnings of such an information-
03300	processing theory, and that the time is ripe for an effort to integrate
03400	computational ideas and empirical investigation. In attempting 
03500	to further this suggestion, we outline what we think to be some of the
03600	salient and potentially fruitful concepts which AI has generated; we review
03700	what we  take to be some pertinent investigations which have appeared in the 
03800	psychological literature; and we attempt to gather these two threads together
03900	to begin weaving the fabric of a coherent information-processing approach to
04000	the perception of the visual world. It would be wise at this point to make clear
04100	our concern over the problem raised by the different hardware structures 
04200	available to artificial and natural vision systems: our arguments will stress
04300	commonality of computational processes, rather than any features of the
04400	implementation of these processes, whether in a digital computer or the brain.
04500	We think it worthwhile, however, to devote some effort to suggesting how
04600	processing which may be essentially serial in an artificial system may have its
04700	nature altered when it is considered in the context of the (at least partly)
04800	parallel mechanisms available to the brain. We do not subscribe to the view
04900	that a simple isomorphism must exist between the distinctive features of serial
05000	and parallel computations, at least at the level which concerns us here.
05100	
     

00100	.next page
00200	.once flush left
00300	%3 2. DEFINING THE PROBLEMS.%*
00400	
00500	In this section we attempt to set out what we think to be the major problems
00600	to whose solution  an information-processing theory of perception should
00700	aspire. A large number of these problems stem from a realisation that a large
00800	part of perception is %2not%1 concerned with %2recognizing%* objects which may already
00900	have appeared in the repertoire of our experience, but rather with the %2description%*
01000	of scenes. That is to say, the problem is that of transforming a large body of
01100	sensory data, for example, intensity, colour and texture at a large set of points
01200	within the visual field, into a more compact description or representation of
01300	the scene being viewed, so that this description may be incorporated into,
01400	and interact with, the large corpus of information, both visual and non-
01500	visual which may already be stored in memory. Our claim is that this view
01600	of the nature of perception, non-controversial though it may seem at first 
01700	glance, immediately reduces the usefulness of a large body of techniques, 
01800	historically labelled `pattern classification',and largely based on template
01900	matching, which were
02000	developed for recognition of simple geometric patterns, for example, alphanumeric 
02100	characters. Furthermore, it presses upon us the necessity for representational
02200	formalisms which capture in some smooth way the whole range of sensory and 
02300	symbolic information to which humans have access.
02400	
02500	Several questions must immediately be asked of this data-description process:
02600	
02700	(a) just how much of the sensory data are discarded during the analytical 
02800	process? 
02900	
03000	(b) when are they discarded? We will show that they cannot be thrown away too soon,
03100	but must be kept available for a more detailed analysis at any stage;
03200	
03300	(c) are the data discarded at all, in fact, or do we remember images for 
03400	long periods? If not, just what form does the representation in long-term
03500	memory take which allows the reconstruction of the mental images with which
03600	we are subjectively familiar? Is the representation purely pictorial in nature,
03700	or purely symbolic, or does it have some of the characteristics of both?
03800	Later in this discussion, we will be concerned to defend the view that 
03900	any distinction between the pictorial and symbolic features of memory is 
04000	a functional one, rather than a reflection of the true nature of the internal
04100	code.
04200	
04300	We approach our task initially via a fairly detailed examination of some
04400	possible stages in the processing of visual information: what kinds of data
04500	are needed at the various stages?; what can each stage achieve on its own?;
04600	what help is needed from higher level processes? Our suggestion is that a 
04700	strictly hierarchical structure, such as might be inferred for instance from
04800	the original ideas of Hubel and Wiesel(Hubel & Wiesel,1968),
04900	 is not appropriate for the task in hand.
05000	A visual system which consists of a discrete set of analytical stages, each
05100	encoding its own specialized conclusions for communication to an immediate
05200	superior, cannot achieve the kinds of computations necessary for the 
05300	understanding of visual input. The alternative structure is %2heterarchical%*, 
05400	which is to say that there is not strict linear ordering of computational
05500	steps. In a heterarchical system, information flow can be bi-directional -
05600	advice and queries from more proximal stages can influence the behaviour of
05700	more distal stages. We might note in passing that such an organization seems to 
05800	be implied by several neurophysiological studies which have recently come to 
05900	hand. Horn and Hill(Horn & Hill,1969) demonstrated gravitational (presumably 
06000	vestibular)
06100	effects on the properties of visual orientation detectors in cat striate cortex;
06200	Spinelli and his colleagues(Spinelli,Starr & Barrett,1968),
06300	 in a series of studies, have demonstrated
06400	an influence of auditory and somaesthetic stimuli on the shapes of retinal
06500	receptive fields; and other studies have shown
06600	changes in retinal receptive field sizes with changing activity in the
06700	oculo-motor system (accomodation).
06800	
06900	We should also take note here that several recent experimental studies have suggested that
07000	the hierarchical mechanisms proposed by Hubel and Wiesel to account for the
07100	apparent progressive specialization of properties amongst visual cortical
07200	cells (from simple to complex to hypercomplex) may not be correct. Hoffman and
07300	Stone(Hoffman & Stone,   )in a study of correlations between receptive-field
07400	 properties
07500	and conduction velocities of such cells found, firstly, that at least 40 percent 
07600	of complex cells are activated monosynaptically by fast afferent fibers with delay
07700	times of 1-2 msec., while simple and hyper-complex cells are not activated 
07800	mono-synaptically by fast afferents, but some proportion is activated mono-
07900	synaptically by slow fibers (3-5 msec.). Secondly, they have shown that fast
08000	and slow fiber activity is relayed separately in the lateral geniculate nucleus.
08100	They have suggested that simple, complex and hypercomplex cells process
08200	visual information in parallel rather than in the serial manner proposed by
08300	earlier workers.
08400	In later remarks, we will suggest a similar
08500	parallel processing model; furthermore, we will suggest that the differing
08600	activities of the fast and slow pathways may be correlated with differing
08700	features of the representational system.
08800	
     

00100	.skip 
00200	We actually want to make the strong claim that visual analysis of scenes such as
00300	we see in every-day life is much more difficult than one might at first assume,
00400	and that the very richness of the information available to us imposes significant
00500	constraints upon the nature of an analytical system which is able to cope with
00600	it. Consider some of the simplest problems:
00700	
00800	(a) the volume of data: at present the Hand-Eye project at Stanford uses for
00900	its visual sensor a television camera which generates data consisting of a 
01000	four-bit (i.e. 16 grey-level) description of light intensity at each of about
01100	330 by 260 points. The camera generates these data at about one scan every 60th.
01200	of a second,but it takes far longer to analyse it even to the fairly primitive
01300	level that we have achieved so far. The application of an edge-operator to 
01400	each of these 80,000 points takes a minimum
01500	of 150 microseconds/point on our present machine. When one 
01600	realizes further that this volume of analysis may have to be done on each of 
01700	several stereo pictures (for depth by correlation), with colour, it is easy to
01800	see that processing time can get out of hand quite rapidly.
01900	Compare this with the human
02000	visual system, which has about an order of magnitude more resolving power, and yet 
02100	manages to do much more in a small fraction of a second. Naturally, the human
02200	system has much more specialized hardware, but we hope that the point will 
02300	emerge that even so the data volume problem is considerable;
02400	
02500	(b) noise: the human eye is nothing like a perfect camera, but has considerable
02600	optical and electrical defects, anisotropies in resolution and abberation effects,
02700	for example, which must be overcome by suitable noise-reduction
02800	techniques such as optimal sampling methods and comparison of computations in
02900	neighboring pathways;
03000	
03100	(c) the eyes are not fixed: they move both with the rest of the body frame and
03200	independently in their orbits. Why is it then that the visual world appears
03300	to be stablised under these movements? Helmholtz (Helmholtz,1963) pointed out that 
03400	passive movement of the eyeball causes a disconcerting shift in the perceived
03500	field, and Brindley and Merton(Brindley & Merton, 1960)
03600	 showed that the apparent stability is not
03700	due to feedback from proprioceptors in the orbital muscles themselves, but rather
03800	to correction for the movement at a much higher level, involving direct 
03900	interaction of the oculo-motor and visual centers;
04000	
04100	(d) variable view  point:  three  dimensional  objects,  unlike
04200	characters  on  a  piece  of  paper,  can  present  many  and  varied
04300	appearances. Because of the fact that we are separated  from  them  in
04400	three-space,  distortions  are  caused by the phenomenon of %2perspective%*.
04500	If we are are going to design a visual system , we have to have  some
04600	fairly   sophisticated   way  of  recognising  objects  from  various
04700	viewpoints, and even making  predictions  about  the  appearances  of
04800	parts  of  objects  that  we can't see. That is to say, we must have an
04900	efficient way of mapping the visual  appearance onto descriptions which we
05000	already have stored. We have to develop  ways  of  dealing  with
05100	transformations   which   are  translational  and  rotational,  and
05200	simultaneously deal with scale changes as we approach or recede from
05300	objects. 
05400	For an approach
05500	based on matching features from templates in any simple-minded way,
05600	in a three-dimensional world, the obscuration of one object
05700	by another yields yet another fatal difficulty.
05800	Just how difficult and subtle problems of  three-dimensional  visual 
05900	geometry are  is  suggested  by the fact that it was not until the Renaissance
06000	that artists (da Vinci,1956;Alberti,1547) were
06100	able  to  formulate rules for the naturalistic representation of space in
06200	painting. Painting  and  art in general raise another issue, which is
06300	the inverse of the everyday scene one - that of the cartoon.  Why  is
06400	it  that  so much visual information can be conveyed by a few strokes
06500	of the brush or pen? What are the mechanisms which allow us  to  fill
06600	in?  Art reveals most clearly the issue of interpretation of pictures
06700	(see for example the excellent discussion by Gombrich(Gombrich,1960). Remember
06800	the duck/rabbit cartoon(Wittgenstein,1953), and Boring's Young Woman/Old Hag
06900	(Boring,1942)
07000	drawing, as well as examples of figure-ground  confusions which are all aspects
07100	of the  very  difficult  question  of  multi-stable  states  in
07200	vision,  which  crops up both in depth-perception and in interpreting
07300	line drawings as three dimensional objects. Why does the Necker  Cube
07400	apparently  have %2two%* stable three-dimensional interpretations,  as  well  as the
07500	perfectly simple two-dimensional one?  What  are  the  factors  which
07600	decide  which  of  the  stable  states  we  land in, and why is it so
07700	difficult to switch from one state to another?
07800	
07900	Actually we hope that  it will become clear later that the three-
08000	dimensionality of the world is a help  rather  than  a  hindrance  to
08100	scene  analysis. Techniques have been developed(Nevatia & Binford, 1973)
08200	for  using  depth  information alone for the segmentation of scenes containing
08300	curved objects and these same workers have proposed a volume-based representation for such objects.
08400	We discuss this in more detail in Section 5;
08500	
08600	(e) variability in lighting: light can  be  of  many
08700	colours  and intensity, and come from one or more sources, as well as
08800	being reflected off some surfaces onto adjacent ones. This again is a
08900	source  both  of annoyance and information. It's annoying that one has
09000	to keep changing the characteristics of one's sensory  mechanism  all
09100	the  time  -  one has to adapt for varying light levels. On the other
09200	hand, if one knows something about the lighting situation one can use
09300	information from shadows and shading to reconstruct some aspects of the shapes of
09400	objects. A later section will discuss some of the problems which arise in 
09500	judging the colours of object under these
09600	kinds of difficulties. That too will an example where  there  are  some
09700	quite  good  information  processsing  models  which are borne out by
09800	psychophysical data.
09900	
10000	
     

00100	Now that we have laid out some of the difficulties, we should comment on
00200	what we take to be the correct and incorrect methods for coping with them.
00300	The basic thrust of our argument will be that that naive, brute-force methods
00400	such as those developed in the field of pattern classification for recognising
00500	printing,hand-writing, bubble-chamber photographs and so on just will not do
00600	at all for analysing the kinds of scenes with which humans are normally 
00700	confronted.
00800	There seem to be some very striking parallels here  with  attempts  at
00900	machine  understanding  of  natural language. At the beginning of the
01000	sixties, people in that field, and linguists of the  MIT
01100	School, e.g. Chomsky, Fodor , Katz and their colleagues,
01200	thought that the  way  to  describe  natural  language  understanding
01300	systems (whether machine or human) was to write sophisticated context
01400	sensitive generative grammars of one kind or another.It  soon  became
01500	clear  that this
01600	purely syntactic way of going about things did not seem to work.
01700	Language  is  not  wholly a system of rewriting rules applied to base
01800	symbols to produce terminal strings.  It  is  instead  a  system  for
01900	conveying  information.  And  information is not measured in terms of
02000	bytes: its significance (its content) is  very  dependent  upon
02100	the  circumstances  surrounding  its  utterance  and reception. It is
02200	dependent upon a wider kind of context than the  transformational
02300	grammarians  had in mind  -  the context is the state of the
02400	world pertaining at the time somebody says something; and this  state
02500	of  affairs  can  involve  all  kinds  of  non-linguistic factors. So
02600	recently there has been a  much  more  serious effort  to  develop  truly
02700	%2understanding%* language  processors(e.g.Winograd,1973).  Exactly this kind
02800	 of historical
02900	pattern has been apparent in the psychology of  visual perception,
03000	but over a  longer
03100	time  scale.  It may seem paradoxical in view of their well known 
03200	antipathy that, loosely speaking,
03300	the late 19th. century behaviourists  were  akin  in  their
03400	spirit to the generative syntacticists, in stressing the 
03500	purely atomistic structural features of their respective phenomena . 
03600	The Gestaltist reaction went
03700	to the opposite extreme of stressing the wholistic features of perception 
03800	to the detriment of the possibility of partitioning the perceptual process 
03900	into pieces suitable for empirical investigation.
     

00100	.next page
00200	Before going further, we must take some pains to lay out a conceptual
00300	distinction concerning the notion of `level of analysis'.
00400	We have already introduced the notion of a heterarchical
00500	system without defining what we mean by a `level' or `stage' within such a 
00600	system. There seem to be three useful ways of formulating such a definition:
00700	
00800	the first involves a dependence on the  type of data which a process accepts as
00900	its input. According to such a criterion, processes dealing with raw intensity
01000	data (retinal receptors,bipolars, etc.)are at the very lowest level; those which
01100	deal with edges are at a higher level, but are themselves subordinate to processes
01200	which take more global input (from several regions of the visual field, for
01300	example);
01400	
01500	the second way depends upon the amount of `advice' (i.e.interaction from processes
01600	at its own or a higher level) that a process needs in order successfully to
01700	accomplish its task. By this criterion, the lateral inhibition and Retinex 
01800	processes that we shall shortly describe are at a different level from, for
01900	instance, a region-association process.(In fact, for consistency, we rate 
02000	the level derived by this criterion as inversely proportional to the advice
02100	it needs);
02200	
02300	the final criterion reflects the independence allowed to a process to 
02400	initialize itself independently of instructions to do so from some other 
02500	process.
02600	
02700	In a heterarchical system, as we have described it, the clean notion of level
02800	we have just laid out becomes clouded; a better measure is perhaps the 
02900	informational connectivity of a process, i.e. a measure on the number of
03000	sources from which it derives advice and instruction. Crudely speaking,
03100	the less advice that a process needs, the more likely it is to have an
03200	efficient implementation in a parallel, as opposed to a serial, system.
03300	
     

00100	.next page
00200	.once flush left
00300	%3 3. AN ANALYSIS OF LOW-LEVEL MECHANISMS.%*
00400	.skip 2
00500	In this section, we will look in detail at the kinds of information 
00600	processing which might be thought to occur at some early stages in
00700	the visual pathway; in particular, we ask of each of these methods what
00800	it is that it can achieve on its own, with no guidance from other
00900	processes occurring at the same or higher levels (using `levels' in the 
01000	rather careful sense that was laid out in the previous section). This
01100	analysis will also reflect upon the arguments concerning serialism vs.
01200	parallelism which we brought up in that section. We may find that there
01300	are some processes which can apparently be carried out completely 
01400	at one level (and presumably implemented efficiently in parallel hardware);
01500	while, in looking at other processes, we will conclude that 
01600	they cannot function without information from other stages, and perhaps
01700	in the last instance without non-visual information about the nature of
01800	the world. 
01900	.skip 2 once flush left
02000	%2(a) Edges:%*
02100	.skip
02200	Neurophysiological  studies  
02300	suggest that the mammalian visual cortex 
02400	extracts  %2edges%* from raw data. Objects in natural  scenes 
02500	usually have  well  defined  surfaces.   The retinal
02600	projection of such a scene is a patchwork of  regions  demarcated  by
02700	changes  in  intensity.
04000	On  the  other  hand,  scenes  often
04100	consist  of  possibly  irregular regions differing from each other in
04200	terms of rather microscopic properties, for  example  texture.
04300	Gibson argues(e.g.Gibson,1950)
04500	that texture and gradients  of  texture  are  among  the  most
04600	important perceptual constructs. There is a lot to be said for a view
04700	of approximately this type, and there is the beginnings of 
04800	a considerable corpus of literature in AI
04900	concerning
05000	ways  of  describing and recognising texture. It appears  to be very
05100	hard to come up with criteria for clustering the sub-regions of large
05200	inhomogeneous   regions  according  to  some  similarities  in  their
05300	properties (but see Section 3(c)).
05400	
05500	Nevertheless we must start our discussion with ways of finding edges.  The
05600	fundamental  problem  with  just finding edges in a straight- forward
05700	way, even assuming it can be done, is that one ends up  being  swamped
05800	with  large numbers of possibly quite irrelevant edges, resulting for
05900	example  from  minor  variations  in  the  surfaces  of   objects,
06000	reflections  and  shadows.   Textured  surfaces  can contain enormous
06100	numbers of edges, and whether one is interested  in  finding  all  of
06200	these  edges  is obviously a function of the purpose of the analysis.
06300	It is simply not easy to set  up  a  direct  mapping  from  intensity
06400	discontinuities to surface discontinuities.
06500	
06600	A distinction must be made between the process of finding an
06700	edge and that of subsequently following it:  edge-finding, per se, can be
06800	accomplished in a localized manner, while edge-following requires some
06900	more global operation, perhaps involving knowledge about vertices, etc.
07000	in this sense, the feature-specific cells described by Hubel and Wiesel 
07100	are local edge-finders. As yet, little progress has been made in establishing
07200	possible neural mechanisms for constructing larger scale edges from these
07300	local data.
07400	The basic  principles  of  edge  finding/following   are   rather
07500	simple:  
07600	
07700	(i)  edges  are  usually continuous, so once an edge-finder has acquired an
07800	edge point and has some information about the direction in which the
07900	edge  is pointing, it reduces the search space considerably. So there
08000	is a first requirement - the edge finding operator one uses must have
08100	some  elements  of directionality built into it. Further, since 
08200	gradients  can  be  quite  gradual  and   yet   remain
08300	significant,  one needs an operator of variable size. Such variabliity
08400	of size had its own inherent problem, in that  averaging  over  large
08500	areas  can  effectively  destroy large quantities of useful information;
08600	
08700	(ii) edges on the whole form closed contours  and  so  once 
08800	a part  of it has been found, it  is straightforward to follow around the
08900	closed curve until one returns to the  starting  point.  One  operator
09000	for doing just this was developed by Hueckel(Hueckel, 1971).
09100	This is a variable sized
09200	operator, and is  slightly  detuned  to  ignore
09300	high-frequency  data  -  this  removes  the problem of noise from the
09400	camera, and since most edges are at  least  slightly  blurred  doesn't
09500	usually   result   in   decreased  efficiency  of  finding  intensity
09600	discontinuities. The  algorithm  begins  by  systematically  scanning
09700	across  the  picture  with  slightly  overlapping applications of the
09800	gradient operator. If an  acceptable  line  segment  is  found,  then
09900	horizontal  scanning  is  interrupted,  and the program enters a line
10000	following mode: the  last  found  line  segment  is
10100	extended  in  its  apparent  direction,  and  the operator is applied
10200	there. In this way  the  operator  can  trace  back  to  the  initial
10300	aquisition point, keeping track of where it has been so that
10400	it doesn't scan the same points twice. Obviously, deciding whether to
10500	terminate  a  line-following  or continue searching for an edge which
10600	might not exist is a rather difficult piece of decision making  which
10700	involves having the algorithm know about such things as vertices. The
10800	current edge follower, after having found some line segments, attempts
10900	to fit straight lines to them, and looks around at their intersection
11000	points for evidence of a vertex. 
11100	
11200	A somewhat different  edge operator developed by Horn and Binford(Horn & Binford,
11300	1973)
11400	in a  very  clear sense simulates the phenomenon of %2lateral inhibition%*,
11500	an edge enhancing principle which  operates  very  widely  in  animal
11600	visual  systems
11700	and seems to  be  the  mechanism  underlying  the
11800	centre-surround  organisation  of  ganglion  cells in the retina;
11900	further into the visual system it is implicated in the  structure  of
12000	the feature specific cells that Hubel and others have shown.
12100	This  operator subtracts the local average at
12200	each point , and  then  applied  the  gradient  operator  in  several
12300	directions using this average gradient.
12400	
12500	The Hueckel operator as originally formulated  has  one  fundamental
12600	deficiency - its operation is essentially %2one pass%*,in that it
12700	only looks once at any given region of a picture. Since  the  channel
12800	bandwidth  that  is available in a given picture is quite limited, it
12900	becomes very difficult to arrange the electrical  characteristics  of
13000	the camera so that intensity gradients at all points in the scene are
13100	optimally enhanced. The same kind of problem  also  faces  an  animal
13200	visual system. Given this difficuty of globally optimising the edges,
13300	the natural thing to think of doing is to take more than one look, at
13400	least at parts of the picture. By altering the sensor characteristics
13500	between  these  frames,  one  should  be  able  to  bracket  all  the
13600	significant  intensity boundaries. This is the kind of thing that an
13700	"accomodating" edge finder(Tenenbaum & Pingle, 1971) does. Initially it  scans
13800	the  scene  with  an  operator  of  large radius, and with the sensor
13900	characteristics set coarsely. The first scan therefore picks  up  all
14000	the  salient powerful edges in the picture. The accomodation routines
14100	can then examine the range of intensities found in this way,  and  if
14200	necessary  set  the  sensor  characteristics  much  more finely for a
14300	second look at areas  of  confusion  where  for  instance  lines  meet  or
14400	disappear  unexpectedly, e.g.  at  vertices and occlusion points.
14500	
14600	Even at this low level a test-predict-verify
14700	cycle  is operating.  As it traces objects, the edge follower
14800	builds a data structure describing their topology. For each object  a
14900	block  is  created  and  linked  to a circular list of segment blocks
15000	representing the edge segments belonging to  that  particular  blob.
15100	Each  segment  block  has  pointers  in  turn  to  a  ring  of blocks
15200	containing the coordinates of the individual  points.  When  a  trace
15300	being  followed  intersects  a previously found edge segment, closure
15400	has obviously been acheived, the  horizontal  scan  pattern  is
15500	resumed, and the segments already found are merged appropriately into
15600	the object  lists.  Garbage  collection  is  then  easy  -  when  the
15700	controlling  higher  level  process  decides that it doesn't need the
15800	detailed edge data any more, the edge pointers are replaced by a more
15900	parsimonious  list  of corners and the equations of the lines joining
16000	them. It is easy to see then how a previously detected object occluding
16100	the  edge  currently  being traced can be ignored by the edge tracing
16200	algorithm - all it has to do is extend to line it is tracing  to  the
16300	furthest intersecting edge of this occluding object, and continue the
16400	edge scan from there. This mode  of  working  allows  a  much  better
16500	performance  than  the  older and more simple minded edge followers -
16600	it is much more immune to missing patches of edge data.
16700	
16800	
     

00100	.skip 2
00200	.once flush left 
00300	%2(b) Colour:%*
00400	.skip
00500	Given a receptor mechanism sensitive  at  three  conveniently
00600	spaced  wavelengths, such as is available in the retina 
00700	one  might  think that it would be a relatively
00800	easy matter to arrive at judgements  of  the  colours  of  points  or
00900	regions  within a scene. Unfortunately this is not the case, because it
01000	is very  easy  to  show  that  the  mapping  from  physical  stimulus
01100	space(ie.  that  containing  wavelength  and  intensity) to sensation
01200	space is not an isomorphism. One of the most  elegant  demonstrations
01300	that   the  relationship  between  flux  at  various  wavelengths  of
01400	reflected  light  and  the  associated  colour  sensations is not
01500	straightforward was given by Edwin Land in his William James Lectures
01600	at Harvard(see Land & McCann,1971). His experimental subject was a pastel 
01700	drawing of a street
01800	scene  by  Jeanne  Benton, which included a green awning on one side
01900	and a red door on the  other.Light  at  a  long  wavelength(650nm)was
02000	shone  on to the awning, while light of a middle wavelength(540nm)was
02100	shone onto the door in such a way that the same long wave  flux  came
02200	from  the  centre  of  the  awning  and  the  centre  of the door and
02300	similarly for the medium wavelength. In other words,the  total  flux
02400	incident  on  the  retina  from  the  two  regions  was  identical in
02500	intensity and wavelength mixture. In long wavelength light alone  the
02600	door appeared very light and the awning almost black, and the reverse
02700	was true in middle wavelength light alone. However in the mixture  of
02800	light,  the  awning appeared green and the door appeared red! Land in
02900	his Retinex theory(op.cit) proposed that the colour of objects is  determined
03000	by  their  lightness (the product of flux and reflectivity)
03100	at  three  distinct  wavelengths,  and that the
03200	perceived colours result from the  interaction  of  these  receptors.
03300	Thus  areas  that  are  lighter in long wavelength and dark in middle
03400	wavelength  light %2always%* look  red,  independently  of  the  actual
03500	wavelength  distribution  in  the  reflected  light. Areas that look
03600	light at medium wavelengths and dark in long always are perceived  as
03700	being green.  The  other  result of Land's studies is the conclusion
03800	that the three sets of independent receptors  are  sensitive  to  the
03900	phenomenal  lightness  of  the  regions in the scene, rather than the
04000	actual brightness of the region. That is to say, all the measurements
04100	of  incident light by the retinal receptors are %2normalised%*. This has
04200	many advantages - it allows a deemphasising of  shadows  and  gradual
04300	brightness   changes   across  uniform  coloured  regions,  and  also
04400	automatically corrects for colour casts, since the colour of any  one
04500	region  is  not  judged  absolutely,  but  relative to the perceived
04600	colours of its near neighbours.  A program has been developed by one of
04700	us which  simulates this model of colour analysis(Thomas,1974). It looks at
04800	the scene through three filters, and uses the  normalised  intensity
04900	data  at  these wavelengths to construct a colour triangle:
05000	it  uses the  extremes  of
05100	intensity  at each wavelength to describe the vertices and define the
05200	white point. This program has had some success in dealing with lighting
05300	situations involving colored shadows and other difficult casts, and has
05400	been found to pass at least some standard tests for anomalous color vision.
05500	
05600	Horn(Horn,1974) and Marr(Marr,1974) have investigated the way in which the
05700	Retinex operation which we described may be implemented in the human
05800	visual system, and the latter has suggested that one function of
05900	the retina is the computation of this retinex lightness function along
06000	four channels (rod and 3 coloured cone) simultaneously. 
06100	
06200	The Retinex operation is clearly well-adapted to being carried by 
06300	special parallel-processing hardware, being essentially a one-level
06400	process, independent of any particular knowledge about the scene being
06500	looked at.
06600	
06700	
     

00100	.skip 2 once flush left
00200	%2(c) Texture:%*
00300	.skip
00400	On the whole far less effort  has  been  devoted in AI  to  outdoor
00500	scenes  than  to  toy  scenes, probably because the problems are more
00600	difficult in the former. What does seem clear  is  that  schemes  for
00700	handling  natural  scenes  will depend far less on edge following and
00800	far more on analysis of depth and texture. Texture is a  particularly
00900	difficult question.
01000	
01100	First of all textures can occur at different levels within  a
01200	scene.  Secondly,  it  is  very  difficult  to arrive at satisfactory
01300	descriptors of textural features. That texture and related components
01400	are  important in human vision is obvious from  studies on the
01500	frequency characteristics of the  visual  system.  Campbell and
01600	others (e.g.Campbell&Robson,1968) have demonstrated psychophysically that 
01700	visual thresholds
01800	are directly related to  the  spatial  frequency  components  of  the
01900	stimulus  field.  They  also  showed  that there are cells in the cat
02000	striate cortex which are highly selective for the  spatial  frequency
02100	of  gratings over a wide range of frequencies. Pollen(Pollen,  ) demonstrated that the
02200	simple cells described by Hubel and Wiesel are sensitive both to  the
02300	stimulus  area  and to its brightness, the implication being that the
02400	data from a single simple cell  cannot  therefore  provide  a  unique
02500	characterisation  of  a stimulus. He went on to claim that the visual
02600	cortex operates by a technique of  strip  integration  which  can  be
02700	described as an operation in the Fourier domain.
02800	
02900	The Fourier transform model of vision 
03000	which has been proposed by some workers (e.g.Pollen,op.cit)
03100	is a version  of  the  template
03200	matching  paradigm, which holds that  the  primary  task  of  a visual system is
03300	recognition, i.e. matching  an  image  (of  an  isolated  object)  with
03400	templates  from previous images.   The templates could be portions of
03500	images or quantities derived from images, quantities such as  moments
03600	or Fourier coefficients. The set of problems motivating this paradigm
03700	is classification of isolated, two-dimensional forms  among  a  small
03800	set  of  possibilities:  character recognition is typical. We have contended
03900	that template matching is inadequate for the visual requirements of a
04000	human; unfortunately, the Fourier model is not very effective even for template
04100	matching.  A model for template matching  is:  portions  of  previous
04200	images  are  matched  against  similar portions of the current image.
04300	But what part of the original image should be taken as a  template?    It
04400	appears that a primary part of the visual system is
04500	a system capable of segmentation rather than a template  matching  mechanism.
04600	The  templates  could  be  supplied  by  revelation, or inferred from
04700	another (non template matching) facility  such  as  motion.   In  the
04800	paradigm,  the  chief  difficulty  is  the  computational  effort  in
04900	matching  templates  to  scenes  involving  rotation,  dilation   and
05000	translation  of objects.   We contend that real world visual problems
05100	involve articulation, obscuration, and  judgments  of  similarity  of
05200	objects  which  are  arbitrarily  different  according to non-trivial
05300	metrics of template matching:  the class of cups is not identified by
05400	any  unique  global  shape  or  by  enumeration,  but  by  being open
05500	containers (capable of holding liquid) of  a  certain  volume.  Thus,
05600	similarity  depends  on  a  description  of  three-dimensional  form,
05700	similarity  judgments  often  also  depend  on   the   facility   for
05800	segmentation,   and  local  description.  Even  within  the  template
05900	matching  paradigm,  the  Fourier   transform   model   has   serious
06000	difficulties.   The  Fourier  transform is equivalent to the original
06100	image; the model is useful only if there are  great  simplifications.
06200	The  supposed  advantage  of  the  Fourier transform is translational
06300	invariance. However, the transform is translationally invariant  only
06400	for  periodic  functions.   Objects are finite and images are finite.
06500	In  this  case,  the  transforms  are  position  dependent.   Another
06600	difficulty  is  that if  there  are  several  objects  in a scene, the
06700	Fourier transform depends on  all  of  them.   The  spatial  template
06800	matching at least allows some localization to the template.
06900	Experiments   show   that  linear  systems  analysis  is  useful  for
07000	describing  response  of  the  visual  system  to  various   stimuli.
07100	Further,  there  appear to be separate channels with about one octave
07200	width.  The alternative, that the system  acts  as  a  single  linear
07300	filter,  seems  like  a  straw man set up only to be knocked down, in
07400	view  of  the  wide  range  of  tasks  facing  the   visual   system.
07500	Experiments show that the detection threshold for square waves can be
07600	predicted adequately from the component of the fundamental frequency.
07700	Further,  square  waves  are  distinguishable from sine waves only at
07800	contrasts such that the third  harmonic  is  above  threshold.  These
07900	results  do  not  discriminate  against  an edge detection mechanism,
08000	however, since the ratios of sensitivities for square  wave  to  sine
08100	wave   would   be  very  similar.  and  whatever  means  is  used  to
08200	discriminate between the two stimuli must  give  similar  results,  i.e.
08300	must  be  dominated by the third harmonic. Any edge detector of fixed
08400	size must have a frequency response with approximately one octave width.  If
08500	for  example, the spatial weighting is unity across the detector, the
08600	response is zero for sine  waves  with  period  half  the  width  and
08700	response is half for sine waves with period twice the width.
08800	
08900	In the light of these misgivings and apparent inadequacies, we must take
09000	some pains to separate out those features of a Fourier-domain which are useful
09100	and those more generalized ones which have little support, either theoretical
09200	or experimental.
09300	Bajcsy (Bajcsy,1972) suggested  a  region  growing  algorithm for analysis of
09400	outdoor scenes which was based on texture description. This attempted
09500	to  deal  with  problems  of  pattern  grouping  and proximity in an
09600	n-dimensional feature space. Some of the nicest studies of  proximity
09700	have  been  those of Julesz(Julesz,1971) and of Shepard(Shepard,1964),
09800	Kruskal(Kruskal,1964) and others. Shepard
09900	and his colleagues have developed some powerful  scaling  techniques
10000	for  extracting important stimulus dimensions from data about sensory
10100	judgements, and Julesz applied  them
10200	to judgements of similarity of visual textures; his finding was that the
10300	most dominant features  were  brightness(contrast)  and  orientation.
10400	Bacsy's  work  attempted to use some techniques in the Fourier domain
10500	to describe such important features, and derived a way of mapping the
10600	information  from  the  power  spectrum  of  a  scene  into  textural
10700	properties in the spatial domain. Obviously the power spectrum  of  a
10800	scene  is  invariant under translation, but not under rotation, so it
10900	provides a way of specifying directionality. The phase  spectrum  on
11000	the  other hand can be used to specify position in the scene.  She
11100	was able to show how such a powerful set of descriptors  for  texture
11200	could be constructed and used in an algorithm for region growing. The
11300	decision problems are quite  difficult,  since  what  is  essentially
11400	involved  in  expanding  regions is a large number of judgments about
11500	proximity in n-space. The sheaf representation that she  used  allows
11600	one to pass from local judgments to more global structures, the whole
11700	process being imbedded in a hypothesis making/verification  paradigm.
11800	Her program was able to extract significant textural information from
11900	outdoor scenes involving trees, water, grass and so on, and use  this
12000	information to segment the scene in natural ways.She also
12100	devoted  some  thought to the
12200	significance of texture gradients in the estimation of
12300	depth, along the lines suggested by Gibson (op.cit)
12400	
12500	
12600	
12700	
12800	
     

00100	.skip 2 once flush left
00200	%2(d) Depth:%*
00300	.skip
00400	The measurement of depth is another good candidate for a process which may
00500	essentially be carried out in parallel with little information other than
00600	the raw intensity data from two separate viewpoints a small angle apart.
00700	The pioneering work of Julesz(op.cit.)has shown clearly that stereoscopic depth
00800	information may be gained in the absence of monocular features by a bulk
00900	correlation between images from the two eyes.
01000	Of course, in normal vision, many other cues
01100	interact with this straightforward computation (for example, monocular
01200	features, movement parallax, perceived size and interposition). Blakemore,
01300	in an extremely interesting study (Blakemore,1970) has shown that the cat's visual cortex
01400	has a joint feature/depth representation, in that all orientation-
01500	specific columns of binocularly-driven cells are of either a constant-depth
01600	type,viewing a thin sheet of visual soace, a few degrees wide, at a given
01700	distance, or of a constant-direction type viewing a cylinder of visual space
01800	directed towards the inter-ocular axis.
01900	
02000	A program has been written at the Stanford AI Laboratory(Pingle&Thomas,1974)which
02100	carries out
02200	a feature-driven bulk-correlation able to achieve a resolution of about 1mm.
02300	at one metre. It is envisioned that
02400	this program will perform a crude 3-dimensional segmentation of a visual
02500	scene as a preface to further analysis. At present this correlation process is
02600	being carried out on a serial machine; a special-
02700	purpose hardware device which will speed up the computation 
02800	by a factor of 50 to 100 times will shortly be available.
     

00100	.next page
00200	.once flush left
00300	%3 4. THE INTERMEDIATE LEVEL.%*
00400	.skip
00500	In this section  we move on to consider the possible behaviour of a vision
00600	system which has been provided with the kinds of information which we 
00700	discussed above. In particular, we will be concerned with separating the
00800	segments of objects from the background and from each other, and with 
00900	their association into coherent three-dimensional bodies. The stress will be
01000	on the wealth of possible partitionings which are possible for any reasonably
01100	interesting scene, and with the enormous computational problems which arise
01200	from a failure to introduce an adequate and systematic semantics for pictures,
01300	where by `semantics' here we mean a set of interpretative rules consonant with
01400	the large body of facts, both intuitive and intellectual, which human observers
01500	bring to bear upon their percepts.
01600	
01700	The psychological  literature  on  perception  of  scenes  is
01800	rather  confused,  and  is  really  concerned only with perception of
01900	characters (alone and in context) and line  drawings  of  3-dimensional
02000	objects; only a very few studies have ever been done on perception of
02100	real world scenes containing  large  numbers  of  objects
02200	against  a complex background(e.g.Biederman,1972; Biederman,Glass&Stacy,1973)
02300	
02400	Some of the earliest and  most  significant  studies  of  the
02500	perception   of   figures  were  those  on  perception  of  retinally
02600	stabilised stimuli, beginning with those of Pritchard et.al(Pritchard,Heron&Hebb,
02700	1969)and  of  McFarland(MacFarland, 1968)
02800	When images of line drawings are optically stabilised on
02900	the retina, so that they do not move relative to the receptor surface
03000	when  the eyes move, the figures are at first perceived normally, but
03100	soon the  perceived  image  begins  to  disintegrate  and  eventually
03200	disappears  altogether.  This  is obviously because of the fatigue of
03300	the receptors. What  is  interesting  is  that  the  figures  do  not
03400	disappear wholistically, but rather the parts disappear independently
03500	- the first things to go are vertices,  followed  by  line  segments.
03600	One  might therefore imagine that vertices and line segments form the
03700	primitive structures in perception. One might go on to suggest that a
03800	good  way  of  studying  the  perception of line drawings would be to
03900	carry out eye-tracking studies, and this was indeed done by  Kaufman
04000	and  Richards(Kaufman&Richards,1969). They  did  show that 
04100	acute-angled vertices are better
04200	attractants of visual fixation than obtuse ones.  However  they  also
04300	showed  that in figures subtending less than 10 deg. of visual angle,
04400	very little scanning if any is done. Instead the eyes tend to  fixate
04500	rigidly upon some point within the envelope of the figure. To be more
04600	exact, fixation was predominantly upon that point  which  would  form
04700	the  centre of gravity of the object, %2were it three-dimensional%*. 
04800	Even at this primitive stage within the visual process, prejudices about
04900	the interpretation of the line drawing are already affecting the
05000	perceptual mechanism.
05100	
05200	Other studies have examined the notion  that
05300	three dimensionality and simplicity are two organisational principles
05400	which  are  somehow  inherent  in  the  visual  process.   These  two
05500	principles  are  but  facets  of the old Gestalt %2Praegnanz%* law, which
05600	said  that  scenes  are   perceived   according   to   the   simplest
05700	interpretation  which  is  compatible with the sensory data. In 1953,
05800	Hochberg and McAlister(Hochberg&McAlister,1953) conducted some 
05900	experiments on  the  perception
06000	of  line drawings representing a  cube in various orientations (due
06100	originally to Kopfermann). The clear cut result of  their  study  was
06200	that  the  figure possessing  the  best  symmetry as a %2two%* dimensional
06300	pattern was %2least%* often seen as a %2cube%*.(drawing a  in
06400	Figure  1.) This leads to a connection with the rules 
06500	of symmetry  much discussed by the Gestalt psychologists. Consider the kinds  of
06600	organisational  groupings that  are going on in figure 2(Attneave,1968).
06700	The case of
06800	the clusters of triangles is a very interesting one -  considered  in
06900	isolation,  any  one of these triangles can appear to point in one of
07000	three directions. But when considered as clusters , all the triangles
07100	in  a  group  appear to point in the same direction (even though this
07200	dominant direction changes from time to time).  The  conclusion  from
07300	phenomena  such  as  these  is  that  the  perceptual  system  cannot
07400	simultaneously employ more than one axis of symmetry. Attneave(op. cit.)
07500	pointed  out  that when the triangles are viewed from an acute angle,
07600	they optically become isosceles, but  perceptually  they  keep  their
07700	ambiguity of direction. He suggested that the reason for this is that
07800	competition between the various axes of symmetry is going on  not  at
07900	the  level  of  the  retinal projection but rather within an internal
08000	model of 3-dimensional space. He further suggested that this internal
08100	model may consist of a Cartesian framework, wherein groups of figures
08200	such as those above may be described in two ways:
08300	.begin indent 8
08400		(a) in terms of local (figural axes)
08500	
08600		(b) in terms of the difference between the figural  axes  and
08700	the  general  axes of the visual field.
08800	.end 
08900	The Praegnanz principle would
09000	then dictate that the description chosen in a particular  case  would
09100	be  that which is simplest in terms of the orientation axes. Attneave
09200	has done some other studies suggesting that we do  indeed  have  some
09300	representation  of  three  dimensionality  in  our  perceptual  world
09400	models. These  involved  showing  that  the  apparent  tridimensional
09500	orientation  of  a  cubical  figure(represented by a line drawing) is
09600	determined by tendencies to make the object as  simple  as  possible.
09700	For  example, a figure such as that in Fig.3 could represent any
09800	of a large number of possible parallelopipeds: however if one assumes
09900	that  this figure is a slanted symmetrical cube, then quite uniquely,
10000	the perceived angles %9a,b%* and %9g%* are all equal and right-angles.
10100	This homogeneity of angles conveys a simplicity to the figure
10200	and   therefore  it  should,  according  to  the  minimal  complexity
10300	principle, be perceived usually as a cube(Attneave&Frost,1969)
10400	
     

00100	Another interesting  example  of  a  possible  organisational
00200	principle operating in the perception of polyhedra has been suggested
00300	by Perkins(Perkins,1968) in his analysis of the possible configurations that may be
00400	assumed  at cubical corners. Some combinations of three lines meeting
00500	at a vertex look like corners of cubes,
00600	.figure|4|
00700	while others do not.  Why  is  that?  Obviously  this  phenomenon  is
00800	governed  to  some  extent  by context, but there is also an internal
00900	principle operating. It seems a reasonable  empirical  conclusion  to
01000	say  that  `three  lines  meeting  at  a  vertex  form  an acceptable
01100	representation of a two-faced  cubical  corner  if  and  only  if  it
01200	contains  2 angles less than or equal to 90 deg. whose sum is greater
01300	than or equal to 90 deg.'. We can then  calculate  which  three  line
01400	configurations  form  viable  corners. 
01500	
01600	This  little  study by Perkins is a foretaste of the things we will
01700	talk about next: attempts by designers of artificial vision systems to 
01800	derive
01900	systematic rules for segmenting scenes, particularly those containing
02000	configurations of regular polyhedra. We hope that  
02100	corollaries with the psychological
02200	evidence that	has just been presented will become evident.
02300	
02400	The earliest work in this field was done  by   Roberts(Roberts,1965)
02500	who concentrated on segmenting and recognising scenes containing only
02600	blocks of various  regular  shapes.The  input  was  in  the  form  of
02700	perfect(ie. noisless) line drawings.
02800	
02900	Adolfo Guzman's work(Guzman,1968) was a considerable advance  ,  since  it
03000	was the first attempt to impart some meaning to the parts of pictures
03100	(meaning in terms of the relationships between parts). This  approach
03200	mitigated  the problems with the template matching approach which was
03300	underlying in Roberts' program. Guzman's program  is  two-pass:  on
03400	the  first  pass  it  gathers local evidence about vertices ; the
03500	second pass attempts to use this  evidence  to  achieve  a  plausible
03600	segmentation  of  the  scene.  Again  the  input was a noiseless line
03700	drawing.  The  local  evidence  was  based  on   certain   heuristics
03800	concerning possible configurations of plane regions at vertices.
03900	.figure|5|
04000	Each  type  of  possible  vertex  contributes  information about the
04100	relations between neighbouring regions.  The  "arrow"  link  provides
04200	evidence  for  some connection between the two regions on either side
04300	of the shaft. The "fork" configuration provides evidence  for  linkage
04400	between  the  three  contiguous  regions.  The  "TEE" junction offers
04500	evidence of occlusion.
04600	Internally,the
04700	relationships between faces is represented symbolically by associated
04800	lists.  At the lowest level, any two regions with a link between them
04900	are considered to be part of the same body. 
05000	This  can  lead  to trouble when fortuitous coincidences occur: the 
05100	answer to this liberality of region association is  to  require  that
05200	there  be two links between the regions before they are considered to
05300	belong to each other. Some inhibitory heuristics were also used,  for
05400	instance,  there  had  to  be compatibility of interpretation between
05500	vertices belonging to the two ends of a given line. Guzman's  program
05600	worked  quite  well on scenes which contained large numbers of bodies
05700	in arbitrary configurations, but was  bothered  by  holes  and  other
05800	non-convexities.  Further,  and  more  important, it was very poor at
05900	segmenting pictures which were missing data, e.g. line-segments. 
06000	Falk(Falk,1970) approached  this  problem  by  using Guzman type heuristics for
06100	associating %2edges%* rather than regions. He was the able to  use  the
06200	verification   methods   we  discussed  earlier  in  connection  with
06300	edge-following to predict the position of lines in the scene: here is
06400	a  classical  example  of  the  kind  of methodology that we have been
06500	advocating - a (fairly)  high-level  program  concerned  with  analysing
06600	regions into bodies is able to induce the lowest levels of the vision
06700	process to look again for edges in sensitive and  crucial  locations.
06800	For  analysing  real  pictures,  this  method  is  quite  superior to
06900	Guzman's.
07000	
07100	Huffman(Huffman,1971) and Clowes have, independently, investigated
07200	in a systematic way the possible configurations of trihedral vertices
07300	seen from a position of non-singular perspective, classifying edges as
07400	%2Convex, Concave%* and %2Obscuring%*.
07500	.figure |6|
07600	Such a  labelling system
07700	immediately imposes quite  stringent  restrictions  on  the  possible
07800	interpretations  of the vertices in this cube. They were able to show
07900	that only six L vertex labellings and three each of  fork  and  arrow
08000	labellings were possible given the kinds of constraints which operate
08100	upon physical objects. In his paper on  Impossible  Objects (op.cit),  Huffman
08200	points out that there is a similarity between the problem of attaching
08300	realisable labels to an arbitrary  picture  and  that  of  parsing  a
08400	sentence  in  a  language:   he  is  able  to suggest that the reason
08500	impossible objects are so difficult to understand is  that  they  are
08600	the embodiments of illegal parsings .
08700	The Huffman-Clowes studies provide  a likely-looking beginning for a
08800	"grammatical" approach to picture analysis. Waltz(Waltz,1972),in his recent
08900	thesis, developed these ideas further : he expanded the set of possible
09000	labellings to  take  into  account  shadows  and  other  vagaries  of
09100	illumination,  and  introduced  a filtering mechanism involving resolution 
09200	of conflicting information from neighbouring edges as well as
09300	a few heuristics concerning legal lighting situations, etc. which 
09400	allowed his program  to  converge  rapidly  onto a plausible
09500	interpretation, rather than carrying out a depth-first search of  all
09600	the  possible  Huffman labellings.
09700	
     

00100	.next page once flush left
00200	%3 5. A GATHERING TOGETHER.%*
00300	.skip
00400	This section will attempt to draw general conclusions from the detailed points
00500	that have been laid out above, will lay out some suggestions for description
00600	schemes for real scenes, and will go on contend  that these arguments have
00700	revelance for the contemporary controversy in cognitive psychology about the
00800	nature of mental representation.
00900	
01000	Given the initial point of view that the major perceptual problem is that of
01100	description rather than recognition, several conceptual difficulties
01200	arise which do not seem to have received the attention we think they deserve,
01300	either
01400	in the psychological literature or in designs for artificial vision systems.
01500	They are firstly that of attention : the problem of deciding at what level
01600	of analysis (in the sense of Section 2) to approach a perceptual task; and,
01700	secondly, that of arriving at representational formalisms which capture in
01800	some smooth way the imaginal and symbolic aspects of human description schemata.
01900	.quote
02000	`By "description" we do not usually mean verbal descriptions; we mean an abstract
02100	data structure in which are represented features, relations, functions, 
02200	reference to other processes and other information. Besides representing things
02300	and relations between things, descriptions often contain information about
02400	the relative importance of features to one another, e.g. which features are
02500	to be regarded as essential and which are merely ornamental.' (Minsky&Papert,1972)
02600	.end select 1 skip
02700	The attention problem is intimately bound up with the particular goals of
02800	the system at any given moment. For example, in driving a car, one's attention
02900	is normally devoted to the road directly in front, but any unexpected 
03000	motion in the peripheral visual field receives immediate notice. In AI, this
03100	behaviour has been captured by the notion of a `demon', i.e. a high-priority
03200	interrupt process which, when activated by a specific event (e.g. motion) can
03300	stop whichever computation is presently in progress and cause some other 
03400	process to be run to deal with the interrupt. This would seem to be
03500	easily implementable in a natural vision system where 
03600	problems of scheduling are reduced by the inherent parallelism. Most attempts
03700	at the development of artificial vision systems have devoted their efforts to
03800	extremely detailed analysis of the whole visual field, involving the actual
03900	recognition of every visible object, usually by matching against some proto-
04000	type which has been learned and stored in memory. This prototype-matching 
04100	matching approach, which is in some ways a generalization of the template-
04200	matching techniques which we scorned earlier, also fails to meet our second
04300	criterion of adequacy, in that it is not easily expanded to a smooth
04400	representational formalism: it is overly concerned with localized features,
04500	at the expense of understanding larger-scale characteristics of a visual scene.
04600	
04700	We will discuss three
04800	approaches which begin to escape these apparent defects: the first, described by
04900	Binford (Nevatia&Binford,1973),involves a generalised volume description of 
05000	curved objects;
05100	the second, due to Winston(Winston,1970) is the beginning of an attempt to describe the
05200	relations between objects in a scene; and the third is Minsky's recent suggestion
05300	concerning the notion of frames(Minsky,1974)
05400	
05500	A representation useful for the work of Binford et al. with
05600	complex objects was chosen to
05700	satisfy design criteria which are relevant to human perception.   The
05800	representation must be generative, that is,  a  rich  set  of  shapes
05900	should be conveniently generated from local primitives.   For that to
06000	be effective, the representation  should  have  a  segmentation  into
06100	parts,  which  themselves  may  be  formed  of  other parts.  For the
06200	part/whole segmentation to be effective, the primitive parts must  be
06300	naturally  defined  and  computationally adequate.  In this case it was decided
06400	to define  primitive  parts  by  continuity.   This  implies  that   the
06500	primitives  are  volume  primitives and not surface primitives, since
06600	surfaces are discontinuous for  objects  with  what  we  conveniently
06700	think of as a single part (e.g. a block).
06800	
06900	The  representations  depend  upon  segmentation  to describe complex
07000	objects in terms of parts.  The representations are graph-structured;
07100	nodes correspond to parts, while arcs correspond to relations between
07200	parts.   Parts may be compound parts, i.e. graphs  of  the  same  form.
07300	Relations  include  relative  position  and  orientation,  degrees of
07400	freedom,  symmetry  and  any  special  knowledge  available.      The
07500	topological  operations  of  cutting  and pasting are used in joining
07600	parts.  Normally, we think of holes as made by cuts and protuberances
07700	as  made  by pasting.
07800	
07900	Primitive  parts are arm-like, described as "generalized local cones".
08000	These parts are described formally in  terms  of  "generalized  local
08100	translational  invariance", appropriate to parts whose cross sections
08200	change slowly along some space curve.  A general cylinder  is  formed
08300	by  an  arbitrary cross section translated along a straight line.   A
08400	cone has a linear variation of cross section along this axis. If  the
08500	scale of the cross section is varied smoothly along the axis, we have
08600	what might be called a local cone. If the cross section is allowed to
08700	vary by distortion or rotation, and the axis is allowed to be a space
08800	curve, then we have "generalized local cones". These are the  volumes
08900	swept  out  by  taking  an arbitrary cross section and translating it
09000	along a space  curve,  meanwhile  varying  the  cross  section  while
09100	holding the cross section normal to the path.
09200	
09300	Cross  sections are represented in the same fashion in two dimensions
09400	as objects are in three dimensions.  Again, part/whole segmentation is
09500	crucial. It is required that parts be defined by continuity. Thus
09600	the parts must be area parts,  not  curve  parts,  since  curves  are
09700	discontinuous  for plane figures which we normally think of as having
09800	a single part.  The primitives are described by a cross section (one-
09900	dimensional) varying smoothly along a plane curve.
10000	Each  element has a local coordinate system.  Each joint contains the
10100	transformation necessary to go from  the  coordinate  system  of  one
10200	element to that of the other.
10300	This process of segmentation allows non-unique representations, and
10400	permits us a  choice  of  simple  representations;  we  can  regard  the
10500	non-uniqueness as an advantage in the light of our comments above about 
10600	the attentional problem.
10700	
10800	The basic philosophy of Winston's work is two-fold:
10900	
11000	(a) that learning is a process of constructing compact descriptions of the 
11100	meaningful relationships between objects in the world; and
11200	
11300	(b) that such description construction is not to be achieved by a statistical
11400	learning paradigm, but rather by judicious teaching, i.e. the choice 
11500	by a teacher of examples which reveal the crucial features of a relation-
11600	description. Very importantly, it may involve exposing the learner to a what
11700	Winston has called the "near-misses": situations which differ from the one 
11800	another in some singularly crucial way.
11900	.figure |7|
12000	Figure 8 shows the kind of data structure  that
12100	Winston's  program constructs to describe the spatial relationships
12200	involved in a simple Pedestal. On of the most  crucial  relations  in
12300	this construction is that of %2support%* - a  feature which is absent
12400	from the Near-miss examples. The role of the teacher in this case  is
12500	to  reinforce  by  means  of  such examples the crucial relationships
12600	which obtain in the scene. Obviously there can  be  a  hierarchy  of
12700	relations which can be exhibited by such examples , some of which are
12800	more crucial than others.
12900	
13000	In this work, and in that of Binford, we think that there are the beginnings
13100	of a viable and fruitful theory of scene descriptions based much more on
13200	topological features than on geometric ones. Developmental studies of
13300	human perception, particularly by Piaget and his colleagues (Piaget &Inhelder,1967)
13400	have
13500	of course revealed that the ability to construct toplogical descriptions
13600	occurs earlier than a more metric and geometric ability. Minsky(op.cit.) in
13700	an extremely interesting paper, has developed the notion of a `frame' which
13800	is essentially a generalization of Winston's scheme particularly 
13900	adapted to revealing at its top level those features of a situation which are
14000	believed to be most important:
14100	.quote
14200	`Seeing is based on  the use of Frame-Systems. A frame-system is a data-structure
14300	for representing a stereotype situation - like being in a certain kind of room.
14400	Attached to this structure are several kinds of information. Some of this
14500	information is about what one expects to happen next. Some is about what to do
14600	if expectations are not confirmed. 
14700	
14800	Collections of related frames are linked together into Frame-systems.
14900	The effects of important actions are mirrored by transformations between the
15000	frames of a system.'
15100	(Minsky,1974)
15200	.end select 1 skip
15300	We do think however that the essentially non-metrical character of Minsky's
15400	frames is somewhat unsatisfactory. His argument seems based on the belief that
15500	`people do not seem to have much metrical ability vis-a-vis three dimensional
15600	imagery'(Minsky, op.cit.);
15700	this does not fit well with the studies of Attneave that we discussed
15800	above, which seem to demonstrate the
15900	contrary.
16000	
16100	This difficulty leads us naturally to discuss the relationship between symbolic
16200	and pictorial methods of representation, and in particular the nature of 
16300	mental imagery. Of late, there has been considerable controversy in the
16400	literature (represented by Minsky's paper discussed above, and by an elegant 
16500	discussion due to Pylyshyn(Pylyshyn,1972)). We will not, in this essay, devote detailed
16600	discussion to this problem: a paper is forthcoming by one of us (AJT) which
16700	argues, %2contra%* Pylyshyn, that mental imagery is a valid phenomenological 
16800	concept, and that while it may be true that the general nature of memory 
16900	structure is that of a symbolic network, it does not therby follow that 
17000	either the subjective experience of imagery or the considerable body of
17100	structural investigation, e.g. by Shepard(Shepard,1971) and especially by Cooper
17200	(Cooper,1973)
17300	on mental imagery tasks involving random three-dimensional forms,
17400	is thereby to be faulted. The argument is basically that the symbolic/
17500	pictorial distinction is a functional and computational one, and that some
17600	form of the dual-code hypothesis advocated, for example, by Posner(Posner,1972) is a 
17700	likely beginning of a satisfactory theory. The major question is whether it
17800	is correct to think of perception as a combination of a distal stimulus and
17900	a mental image thereof, and conversely whether the perceptual machinery 
18000	is active during pictorial imagining.
     

00100	Advocates of a dual-code type of hypothesis, e.g. Bower(Bower,1972) have argued
00200	in favour of a commonality of generation of both pictorial and linguistic
00300	structures. Powerful  and  interesting  evidence for a role of perceptual
00400	concepts in linguistic operations comes from studies of acquisition
00500	of  language  by  children.  Postal's  Universal  Semantic Primitives
00600	Hypothesis (Postal,1966) makes the claim that linguistic  primitives  must  reflect
00700	closely  the  primitive relationships extant in the organism's world,
00800	while Bierwisch (Bierwisch,1969) has made the point that not only must a child
00900	have  such  a  set of primitives, but he must also learn to recognise
01000	them for what they are. E.Clark(Clark,1971) has  suggested  that  at  an  early
01100	stage  a child who is just beginning to use words does not know their
01200	full meaning, but rather identifies them initially with  only  a  few
01300	features of the meaning, which are criterial for its use of the words.
01400	From our point of view here, the crux of her hypothesis lies  in  the
01500	nature of such features:
01600	.quote
01700	`....the first semantic features that a child uses are liable
01800	to be derived from the encoding  of  his  percepts;....  at  a  later
01900	stage,  as  the child learns more about the structure of his language
02000	as a whole, he will  learn  which  percept-derived  features  play  a
02100	particular linguistic role, and which are relatively redundant.'(Op.cit.)
02200	.end select 1 skip
02300	For example studies on over-extension of relational terms are
02400	particularly  interesting: when two words are closely related, e.g.
02500	%2more/less, same/different,before/after%*, the tendency in confusion  of
02600	use  by a child is towards the use of the unmarked member of the pair
02700	in place of both meanings. In the studies by  Piaget and  others  on
02800	over-extended  use  of synonyms eg. boy/brother, there is a suggestion
02900	that the tendency  is  to  use  such  words  synonymously  until  the
03000	relevant discriminating features are learned and added to the lexical
03100	entry. H.Clark(Clark,1971) has carried  this  way  of  thought  as far as to
03200	suggest that an isomorphism exists between the linguistic and perceptual
03300	domains. He proposes that asymmetries seen in the  use  of  pairs  of
03400	polar adjectives e.g. big/small,tall/short, in front/behind are due to
03500	analogous asymmetries which exist in the visual field. It  is  as  if
03600	there  existed  reference  points  within  this  field,  about  which
03700	directions tend  to  be  defined  as  either  positive  or  negative,
03800	depending on their perceptual predominance.
03900	verbal structures.
04000	
04100	
04200	
     

00100	.next page
00200	.once flush left
00300	%3 6. CONCLUDING REMARKS.%*
00400	We hope that, throughout the preceeding sections, the reader has been
00500	able to trace the two skeins of our argument:
00600	
00700	(a) that a vision system, whether natural or artificial, cannot function in
00800	the absence of a two-way flow of information to and from almost every stage
00900	of analysis; and
01000	
01100	(b) that the crucial problem in understanding or designing such systems must
01200	be the formulation of a representational formalism which captures both 
01300	perceptual and non-perceptual information, and allows a smooth transition
01400	between them. We have sketched what we take to be the beginnings of such a 
01500	formalism, but we find that we have to admit that even these beginnings are
01600	at present quite incomplete and unsatisfactory. This incompleteness is in
01700	some measure due to the general failure of the psychological and AI
01800	communities to conduct a coherent dialogue. Our hope is that this paper will
01900	have gone some way towards reducing any difficulty due to terminological
02000	and conceptual differences which exist between the two disciplines,and have therby
02100	inhibited a    potentially fruitful discourse.
02200